Sensitivity Analysis of Simulation Models

This contribution presents an overview of sensitivity analysis of simulation models, including the estimation of gradients. It covers classic designs and their corresponding (meta)models; namely, resolution-III designs including fractional-factorial two-level designs for first-order polynomial metamodels, resolution-IV and resolution-V designs for metamodels augmented with two-factor interactions, and designs for second-degree polynomial metamodels including central composite designs. It also reviews factor screening for simulation models with very many factors, focusing on the so-called "sequential bifurcation" method. Furthermore, it reviews Kriging metamodels and their designs. It mentions that sensitivity analysis may also aim at the optimization of the simulated system, allowing multiple random simulation outputs.simulation;sensitivity analysis;gradients;screening;Kriging;optimization;Response SurfaceMethodology;Taguchi

Validation of Models: Statistical Techniques and Data Availability

This paper shows which statistical techniques can be used to validate simulation models, depending on which real-life data are available. Concerning this availability three situations are distinguished (i) no data, (ii) only output data, and (iii) both input and output data. In case (i) - no real data - the analysts can still experiment with the simulation model to obtain simulated data; such an experiment should be guided by the statistical theory on the design of experiments. In case (ii) - only output data - real and simulated output data can be compared through the well-known two-sample Student t statistic or certain other statistics. In case (iii) - input and output data - trace-driven simulation becomes possible, but validation should not proceed in the popular way (make a scatter plot with real and simulated outputs, fit a line, and test whether that line has unit slope and passes through the origin); alternative regression and bootstrap procedures are presented. Several case studies are summarized, to illustrate the three types of situations.Statistical methods;simulation models

The role of statistical methodology in simulation

statistical methods;simulation;operations research

Experimental Design for Sensitivity Analysis of Simulation Models

This introductory tutorial gives a survey on the use of statistical designs for what if-or sensitivity analysis in simulation.This analysis uses regression analysis to approximate the input/output transformation that is implied by the simulation model; the resulting regression model is also known as metamodel, response surface, compact model, emulator, etc.Regression analysis gives better results when the simulation experiment is well designed, using classical statistical designs (such as fractional factorials, including 2 k-p designs).These statistical techniques reduce the ad hoc character of simulation; that is, these techniques can make simulation studies give more general results, in less time.experimental design;simulation models;sensitivity analysis;regression analysis

Validation of Simulation, With and Without Real Data

This paper gives a survey on how to validate simulation models through the application of mathematical statistics. The type of statistical test actually applied, depends on the availability of data on the real system: (i) no data, (ii) only output data, and (iii) both input and output data. In case (i), the system analysts can still experiment with the simulation model to obtain simulated data. Those experiments should be guided by the statistical theory on design of experiments (DOE); an inferior - but popular - approach is to change only one factor at a time. In case (ii), real and simulated output data may be compared through the well-known Student t statistic. In case (iii), trace-driven simulation becomes possible. Then validation, however, should not proceed as follows: make a scatter plot with real and simulated outputs, fit a line, and test whether that line has unit slope and passes through the origin. Instead, better tests are presented. Several case studies are summarized, to illustrate the three types of situations.verification;credibility;assessment;sensitivity;robustness;regression